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PREFACE 


The  research  described  in  this  report  was  conducted  at  the  Armstrong 
Laboratory,  Human  Resources  Directorate,  Aircrew  Training  Research  Division 
(AL/HRA),  in  Mesa,  Arizona,  under  Work  Unit  2743-25-17,  Flying  Training 
Research  Support.  The  work  was  conducted  under  Contract  F33615-90-C-0005 
with  the  University  of  Dayton  Research  Institute  (UDRI).  The  laboratory  contract 
monitor  was  Ms  Patricia  Spears;  task  monitor  was  Dr  Wayne  L.  Waag. 

This  report  includes  documentation  of  research  performed  on  air  combat 
performance  measurement.  Results  of  the  research  on  modeling  pilot  expertise  in 
air  combat  was  presented  at  the  36th  Annual  Meeting  of  the  Human  Factors 
Society,  held  from  12-16  October  1992,  in  Atlanta,  GA.  Subsequent  research  on 
validation  of  the  model  was  presented  at  the  14th  Biennial  Applied  Behavioral 
Sciences  Conference,  from  6-8  April  1994,  in  Colorado  Springs,  CO.  The 
presentations  were  also  published  in  the  respective  proceedings. 
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The  purpose  of  this  effort  was  to  model  expert  pilot  performance  and  decision  making 
in  one-versus-one  (Ivl)  air-to-air  combat.  Several  knowledge-elicitation  techniques  were  used 
to  extract  air  combat  expertise  from  a  former  fighter  pilot,  who  served  as  the  subject-matter- 
expert  (SME).  Unstructured  and  then  structured  interviews  were  used  to  elicit  the  goals  and 
sub-goals  of  air-to-air  combat,  plus  some  of  the  pilot  behaviors  necessary  to  accomplish  the 
goals.  The  SME  also  flew  a  number  of  combat  sorties  against  another  former  fighter  pilot  in 
the  Simulator  for  Air-to-Air  Combat  (SAAC)  to  demonstrate  pilot  performance  required  to 
accomplish  the  goals  of  air  combat.  Based  on  the  SME’s  verbal  protocols,  a  group  of  air 
combat  rules  were  developed.  A  rule-based  production  system  was  then  designed  to  incorporate 
the  resulting  knowledge  base.  The  production  system  was  also  designed  to  be  capable  of 
analyzing  an  existing  data  base  of  air  combat  engagements.  Expert  system  development  required 
additional  input  from  the  SME  to  identify  specific  values  of  flight  parameters  required  by  the 
production  system.  Upon  completion  and  SME  verification  of  the  expert  model,  it  will  be 
validated  by  comparing  its  performance  to  that  of  our  SME  in  simulated  air-to-air  combat.  If 
the  model  can  successfully  describe  expert  pilot  performance,  the  model  will  be  used  to  provide 
diagnostic  performance  feedback  in  conjunction  with  SAAC  training. 


Air  combat  performance  measurement  research 
has  resulted  in  several  varieties  of  pilot  performance 
measures.  These  include  measures  of  position 
advantage  (e.g.,  McGuiness,  Forbes,  and  Rhoads, 
1984;  Wooldridge,  Kelly,  Obermayer,  Vreuls,  Nelson, 
and  Norman,  1982),  energy  maneuverability  (e.g., 
Pruitt,  1973;  Pruitt,  Moroney,  and  Lau,  1980),  and 
critic^  discrete  events  (e.g.,  Brictson,  Ciavarelli, 
Pettigrew,  and  Young,  1978;  Ciavarelli,  1987).  The 
Armstrong  Laboratory,  Aircrew  Training  Research 
Division  (AL/HRA)  has  developed  and  implemented 
the  Air  Combat  Maneuvering  Performance 
Measurement  System  (ACM  PMS)  which  has  the 
ability  to  collect,  analyze,  and  display  many  of  these 
measures  taken  from  simulated  combat  performed  on 
the  Air  Combat  Maneuvering  Instrumentation  (ACMI) 
range  and  in  the  Simulator  for  Air-to-Air  Combat 
(SAAC)  at  Luke  Air  Force  Base,  Arizona.  AL/HRA 


has  also  validated  some  of  these  measures  (Waag, 
Raspotnik,^ and  Leeds,  1992);  however,  the  measures 
provide  little  diagnostic  value  to  the  individual  pilot  and 
cannot  describe  the  underlying  decision-making  and 
cognitive  processes  used  by  pilots. 

In  an  attempt  to  formulate  diagnostic  measures  of 
air  combat  performance  and  to  better  understand  the 
underlying  cognitive  and  decision-making  processes 
required,  AL/HRA  initiated  a  project  to  investigate  pilot 
expertise  in  one-verses-one  (Ivl),  air-to-air  (A/ A) 
combat.  The  objective  of  the  effort  was  to  develop  a 
performance  model  of  expert  pilot  behavior  in  air 
combat.  The  model  would  serve  as  a  standard  for  air 
combat  performance,  and  actual  pilot  performance  that 
deviated  from  that  standard  would  serve  as  data  for 
diagnostic  measures  of  pilot  performance.  This  paper 
describes  various  procedures  used  to  elicit  air  combat 
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knowledge  from  a  subject  matter  expert  (SME). 
Several  procedures  were  used  to  determine  which 
would  be  most  effective  for  future  knowledge 
elicitation  work.  The  knowledge  base  elicited  by  these 
methods  was  then  incorporated  in  a  descriptive  model 
of  expert  air  combat  performance.  After  more  specific 
values  for  air  combat  parameters  were  obtained  from 
the  SME,  the  knowledge  base  was  coded  in  the  form 
of  a  rule-based  expert  system. 

KNOWLEDGE  ELICITATION  METHODOLOGY 
Subject  Matter  Expert 

The  SME  is  a  former  F-4  pilot  with  combat 
experience.  He  had  been  director  of  the  SAAC  and 
responsible  for  its  operation,  maintenance,  and  upgrade 
to  the  current  F-15  and  F-16  configuration.  The  SME 
has  flown  more  than  500  engagements  in  the  SAAC 
against  pilots  ranging  in  proficiency  from  novice  to 
expert. 

Knowledge  Elicitation 

Several  knowledge  elicitation  methods  were 
employed.  Initially,  the  SME  was  interviewed  to 
obtain  the  top  level  goals  of  air  combat  maneuvering, 
plus  the  subgoals  and  operators  which  allow  the  pilot 
to  achieve  those  goals.  Goals  include:  Place  the 
opponent  in  your  cone  of  fire  while  remaining  outside 
your  adversary’s  cone  of  fire.  Subgoals  include:  Turn 
faster  than  your  opponent  and  point  your  aircraft  in  the 
direction  of  your  opponent.  Operators  to  accomplish 
these  goals  include:  Turn  in  the  direction  of  the  oppo¬ 
nent,  place  him  on  the  plane  of  the  vertical  stabilizer 
(PVS),  and  pull  him  to  your  nose.  A  second  more 
structured  interview  was  conducted  to  identify 
additional  operators  required  to  accomplish  the  goals  of 
air  combat. 

The  SME  also  flew  a  series  of  Ivl  air  combat 
engagements  against  an  another  former  fighter  pilot  in 
the  SAAC.  Each  pilot  began  the  engagements  in  either 
an  offensive,  defensive,  or  neutral  position.  Total  time 
in  the  simulator  was  about  one  hour.  Immediately 
after  simulator  sessions,  the  engagements  were 
replayed  on  the  ACM  PMS.  The  SME  was  videotaped 
as  he  debriefed  his  own  performance  during  replays, 
describing  his  goals,  actions  to  meet  those  goals,  and 
the  necessary  conditions  for  initiating  those  actions. 
This  information  was  used  to  confirm  data  collected  in 
the  interviews,  to  elaborate  and  refine  the  goals  and 
operators,  and  to  determine  which  operators  were 
appropriate  under  what  circumstances.  From  a 
methodological  standpoint,  this  procedure  appeared 
appropriate  to  meet  the  aforementioned  objectives; 


however,  it  was  noted  that  the  SME  occasionally 
confused  air  combat  engagements.  This  was  not 
unexpected  since  his  behavior  was  similar  in  several 
engagements,  and  the  time  between  engagements  and 
debriefing  those  engagements  usually  exceeded  one 
hour.  This  knowledge  elicitation  procedure  may  be 
appropriate  for  identifying  general  principles  of  air 
combat  and  specific  instances  where  air  combat  rules 
apply,  but  may  not  be  ideal  for  investigating  specific 
pilot  decisions  during  air  combat. 

Two  similar  knowledge  elicitation  procedures 
were  then  used  to  add  more  air  combat  rules  to  the 
database.  The  SME  was  audio  taped  as  he  identified  his 
air  combat  goals,  his  actions  to  achieve  those  goals,  and 
the  situational  variables  that  dictated  those  actions.  The 
SME  again  flew  a  number  of  engagements  against  the 
same  former  fighter  pilot  in  the  SAAC  using  the  same 
initial  conditions  as  before.  In  one  condition,  the  SME 
described  what  he  was  doing  and  why,  while  flying  in 
the  SAAC,  and  in  the  other  condition  he  did  so 
immediately  after  flying  each  engagement.  These 
methods  resulted  the  identification  of  a  set  of  situational 
assessment  variables  such  as  nose  position,  closure, 
energy,  aspect,  range,  and  altitude;  specification  of  addi¬ 
tional  air  combat  rules;  and  identification  of  more 
instances  where  the  rules  apply.  It  was  noted  that 
performing  in  the  SAAC  occasionally  interfered  with  the 
SME’s  ability  to  describe  and  interpret  his  actions.  In 
the  other  condition,  where  the  SME  debriefed  after 
engagements,  he  occasionally  forgot  details  of  the  air 
combat. 

A  final  procedure  was  used  to  refine  situational 
assessment  variables,  to  elaborate  and  refine  air  combat 
rules,  and  to  identify  specific  instances  where  rules 
apply  in  achieving  the  stated  goals  of  air  combat.  The 
Sl^  again  flew  a  number  of  air  combat  engagements  in 
the  SAAC  against  the  same  former  fighter  pilot.  In  this 
condition,  opponents  did  not  describe  their  actions  until 
the  end  of  each  engagement.  Engagements  were 
replayed  in  the  SAAC  so  that  opponents  could  see  each 
engagement  recreated  to  include  all  in-cockpit  displays 
and  out-of-cockpit  visuals.  There  is  also  a  freeze 
capability  in  the  SAAC’s  replay  function.  This  allowed 
the  SME  to  freeze  the  replay  so  he  could  fully  elaborate 
on  what  he  had  done  and  why.  SME  comments  were 
again  audio  taped.  As  would  be  expected,  this  condition 
placed  the  least  demands  on  the  SME’s  memory,  and 
with  the  absence  of  the  competing  task  of  flying, 
afforded  him  ample  time  to  describe  the  rationale  behind 
his  air  combat  behaviors.  This  procedure  will  be  used 
in  the  future  to  elicit  knowledge  for  additional  SMEs. 

The  information  obtained  from  the  above 
procedures  was  compiled  and  modeled,  and  then 
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provided  to  the  SME  for  critique  and  review.  He 
provided  additional  detail  to  existing  rules  and 
additional  air  combat  rules  that  were  not  derived  using 
the  previously  described  methodologies.  The  final  set 
of  rules  was  then  prepared  for  inclusion  in  a  rule-based 
production  system. 

PRODUCTION  SYSTEM  DEVELOPMENT 

The  purpose  of  production  system  development 
was  to  provide  a  model  of  expert  performance  in  F-15 
and  F-16  air  combat.  A  requirement  of  the  expert 
system  was  that  it  be  capable  of  interfacing  with  the 
SAAC  database  which  includes  500  engagements 
between  our  SME  and  125  pilots.  The  expert  system, 
named  ISAAC  (Intelligent  System  for  Air-to-Air 
Combat)  was  to  include  a  performance  measurement 
and  critiquing  capability,  and  be  easily  validated  using 
SAAC  engagement  data.  Validation  testing  would 
include  determining  if  the  SME  actually  followed  his 
own  rules,  and  if  the  model  could  predict  air  combat 
winners  and  losers,  and  expert  and  novice  pilots. 

The  initial  ISAAC  model  described  the  first 
several  "moves"  appropriate  for  the  line  abreast  initial 
condition  where  opponents  start  at  co-altitude,  6000 
feet  apart.  The  appropriate  things  to  do,  according  to 
the  SME,  are  to  turn  in  the  direction  of  the  adversary 
until  the  opponent  aircraft  appears  on  the  plane  of  the 
vertical  stabilizer  (PVS)  (a  plane  that  extends  from  the 
nose  to  the  tail  of  an  aircraft),  and  then  pull  hard  on 
the  stick,  so  as  to  point  the  aircraft  at  the  adversary. 
Assuming  both  opponents  do  the  same  thing  (if  one 
does  not,  he  will  be  at  a  significant  disadvantage),  the 
line  abreast  condition  will  convert  to  a  near  head-on 
condition.  If  one  of  the  opponents  achieves  a 
significant  aspect  angle  advantage  as  a  result  of  making 
a  smaller  initial  turn,  that  pilot  should  reverse  the  bank 
of  his  aircraft  to  maintain  visual  contact  with  the  other 
aircraft.  He  should  then  begin  a  lead  turn  before  the 
airplanes  pass  head-on.  This  set  of  actions  will  give 
the  lead-turning  aircraft  an  additional  aspect  angle 
advantage  on  the  next  head-on  pass. 

The  above  example  was  coded  using  the  expert 
system  shell,  PCPlus.  This  shell  requires  production 
rules  that  define  appropriate  aircraft  maneuvers, 
additional  rules  to  define  these  maneuver  rules  in  terms 
of  aircraft  state  parameter  data,  rules  to  search  the 
database  to  determine  if  state  parameters  requirements 
have  been  satisfied,  and  rules  that  report  if  aircraft 
maneuvers  have  been  performed  correctly.  The 
resulting  expert  model  is,  therefore,  a  hierarchy  of 
rules  and  specifications  to  include  goals,  top-level 
rules,  translation  rules,  parameter  definitions,  special 
default  provisions,  and  software  to  process  engagement 


data.  ISAAC  is  primarily  a  backward  chaining 
inference  system  that  proceeds  backward  from  specified 
goals,  chaining  through  the  applicable  rules  to  determine 
if  the  goal  was  satisfied.  Examples  of  goals  include:  a 
maneuver  goal,  to  test  if  pilot  maneuvering  conforms  to 
the  SME  rules,  and  goal  reporting,  to  ensure  that  output 
required  for  future  analyses  is  recorded. 

Top-level  rules  infer  a  yes/no  value  for  each  of 
the  goals.  Maneuver  rules  are  defined  in  terms  of  more 
than  one  specific  pilot  action,  as  described  by  the  SME. 
These  rules  determine  if  a  particular  maneuver  was 
performed  when  the  SME  considered  it  appropriate  to 
do  so. 

Rule  11 

Subject:  Maneuver  1  Rules 

If  :  (Roll_Toward_Bogey_Manl  And  Bogey_On 
PVS_Manl  And  Max_G_Manl) 

Then:  (Tumed_Into_Bogey) 

Top-level  rules  use  terms  which  require  further 
definition  to  determine  values  in  terms  of  available 
engagement  data.  For  example,  statements  such  as  "roll 
toward  the  bogey"  and  "on  the  PVS"  require  translation 
rules  that  specify  the  behavior  that  defines  the  top-level 
rules.  Defining  these  translation  rules  often  required 
additional  interaction  with  the  SME,  since  the  content  of 
these  rules  is  only  implied  by  top-level  rules. 

Rule  23 

Subject:  Translate-To-Data-Manl-Rules 
If:  ((Left_Abreast  And  AroIl_F_2  <  0  And 

Aroll_F_4  <  0)  or  (Right_Abreast  And 
Aroll_F_2  >  0  And  Aroll_F_4  >0)) 

Then:  (Roll_Toward_Bogey_Manl) 

A  third  level  of  the  expert  system  is  parameter 
definitions.  The  parameters  are  used  by  the  translation 
rules  and  must  be  defined  for  the  expert  system. 
Parameters  are  defined  by  such  values  as  number, 
positive  number,  string,  and  single-  or  multi-valued. 
Parameters  also  identify  data  from  the  air  combat 
engagement  file  by  file  name  and  location  in  the  file. 

Aroll  F  2 

Translation:  (Roll  Angle,  A/C  1,  1/4  Betw  Start  And 
CPAl) 

Type:  Singlevalued 

Used  By:  (Rule023) 

Method:  (Dos-File-In  PCFILE.  1 1  Index  58) 

Due  to  the  idiosyncrasies  of  PCPlus,  some 
additional  information  must  be  provided  by  giving 
specific  values  to  parameters.  This  "special  case"  is  an 
output  message  parameter  where  a  value  is  the  message. 
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If  a  yes  value  for  a  yes/no  parameter  or  goal  cannot  be 
established,  PCPlus  will  infer  "don’t  know";  or  if 
unable  to  establish  a  "yes"  value  should  result  in  a 
"no"  value,  then  the  parameter  must  be  given  a  "no" 
default  value. 

Manl 

Translation:  (Maneuver  Segment  1  Appropriate?) 
Type:  Yes/No 

Updated-By:  (RuleOlO  Rule055) 

Used-By:  (Rule033  Rule042  Rule009) 

Default:  (No) 

Software  was  developed  to  pre-process  air 
combat  engagement  data  for  use  by  the  expert  system. 
A  "C"  program  was  developed  to  process  SAAC 
engagement  files  into  a  form  suitable  for  ISAAC.  This 
program  encodes  engagement  file  data  in  the  format 
required  by  ISAAC;  parses  time  history  files  into 
segments  corresponding  to  commonly  occurring  events 
such  as  start,  first  merge,  second  merge,  and  various 
typical  aircraft  maneuvers;  and  provides  computations 
for  variables  such  as  "on  the  PVS,"  "bogey  in  view," 
"percent  of  time  in  cone  of  fire,"  and  percent  of  time 
offensive,  defensive,  or  neutral. 

Subsequent  work  involved  adding  air  combat 
rules  to  the  existing  knowledge  base,  refining  and 
testing  those  rules,  and  partitioning  the  knowledge  base 
into  frames  corresponding  to  commonly  occurring 
segments  of  air  combat.  The  first  frame  includes  rules 
describing  initial  conditions,  rules  required  to  bring  the 
adversaries  to-the  first  merge,  and  rules  that  apply  at 
and  immediately  following  the  merge.  Since  air 
combat  often  involves  a  series  of  near  head-on  passes, 
the  second  frame  includes  rules  that  describe  a  series 
head-on  passes.  Frames  were  also  developed  to 
include  rules  that  indicate  what  the  pilot  should  do 
when  he  is  either  offensive  or  defensive  with  respect  to 
the  adversary.  A  final  frame  was  developed  to  include 
rules  that  apply  when  neither  pilot  is  offensive  and 
both  aircraft  are  in  close  proximity. 

Initial  ISAAC  validation  will  determine  the 
degree  to  which  the  performance  of  the  model 
compares  to  that  of  our  SME.  ISAAC  includes  nearly 
300  production  rules,  but  only  two  dozen  are  primary 
maneuver  rules.  These  rules  specify  a  particular 
course  of  action  that  is  appropriate  for  a  given  set  of 
circumstances  within  offensive,  defensive  or  neutral 
frames.  ISAAC  takes  as  input  pilot  data  from  the 
SAAC  and  outputs  a  report  as  to  whether  or  not  the 
pilot  performed  the  appropriate  maneuver.  To 
determine  the  degree  to  which  SME  performance 
matches  ISAAC  rules,  a  percent  "correct"  was 
calculated  for  each  maneuver  rule  across  the  500 


SAAC  engagements.  A  second  validation  test  was 
performed  to  determine  if  ISAAC  could  predict  air 
combat  winners  versus  losers  and  novice  versus  expert 
performance.  It  was  predicted,  of  course,  that  winners 
and  experts  would  perform  more  in  accordance  with 
ISAAC  than  would  losers  and  experts.  Adversary 
performance  in  the  SAAC,  rather  than  SME 
performance,  was  compared  to  that  of  ISAAC.  Again, 
a  percent  concurrence  between  pilot  and  model  was 
calculated  for  each  ISAAC  maneuver  rule. 

ISAAC  rules  will  be  refined  further  as  a  result  of 
validation  tests,  and  additional  SMEs  will  evaluate  the 
adequacy  and  accuracy  of  the  ISAAC  rule  base.  When 
the  system  has  been  finalized,  it  will  be  integrated  with 
the  ACM  PMS  to  provide  post-mission  diagnostic 
feedback  to  pilots  training  in  the  SAAC. 
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Abstract 

The  Intelligent  System  For  Air-To-Air  Combat  (ISAAC)  was  developed  to 
model  expertise  in  one-verses-one  (Ivl),  within  visual  range,  air  combat. 
ISAAC  incorporates  the  air  combat  rules  of  a  subject  matter  expert  (SME) 
within  a  typical  expert  system  architecture.  Prior  to  implementing  the  system 
in  a  training  environment,  it  was  necessary  to  validate  ISAAC.  To  represent 
expertise  in  air  combat,  such  a  system  should  accurately  portray  the  behavior 
of  the  expert  who  is  the  source  of  the  expertise.  It  is  also  expected  that  pilots 
who  follow  ISAAC  rules  should  perform  better  in  air  combat  than  those  who 
do  not.  To  test  ISAAC,  data  were  taken  from  the  Simulator  for  Air-to-Air 
Combat  (SAAC)  in  which  Air  Force  pilots  flew  Ivl  engagements  against  the 
expert  and  analyzed  by  ISAAC  to  determine  the  degree  to  which  pilots 
followed  ISAAC  rules.  Results  indicate  that  the  expert  followed  his  own  rules 
nearly  80%  of  the  time,  whereas  his  opponents  followed  the  rules  66%  of  the 
time..  All  pilots  followed  ISAAC  rules  more  often  when  they  won  air  combat 
engagements  than  when  they  lost. 


The  Intelligent  System  for  Air-to-Air  Combat  (ISAAC)  is  a  rule-based  production 
system  that  models  expert  pilot  performance  and  decision  making  in  one-versus-one  (Ivl), 
within  visual  range,  air-to-air  combat.  ISAAC  was  developed  based  upon  a  subject  matter 
expert’s  (SME)  verbal  protocols  provided  in  structured  interviews  and  in  protocols  collected 
both  during  and  after  simulated  combat  engagements  flown  in  the  Simulator  for  Air-to-Air 

Combat  (SAAC). 

According  to  the  SME,  the  &st  several  "rules"  appropriate  for  the  line-abreast  setup 
are  the  following:  the  fighter  should  roll  in  the  direction  of  the  bogey  until  the  opponent 
aircraft  appears  on  the  plane  of  the  vertical  stabilizer  (PVS)  (a  plane  that  extends  from  nose 
to  the  tail  of  the  aircraft),  and  then  pull  hard  on  the  stick  to  effect  near  maximum  Gs. 
Assuming  both  opponents  do  the  same  thing  (if  one  does  not,  he  wiU  be  at  a  significant 
disadvantage),  the  line-abreast  condition  wiU  convert  to  a  near-  head-on  condition.  H  one 
opponent  achieves  a  significant  aspect  angle  advantage  as  a  result  of  making  a  tighter  initial 
turn,  that  pilot  should  reverse  the  bank  of  his  aircraft  to  maintain  visual  contact  with  the 
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other  aircraft.  This  results  in  a  lead  turn  before  the  airplanes  pass  head-on.  The  lead- 
turning  aircraft  should  gain  an  additional  aspect  angle  advantage  on  the  next  head-on  pass. 

After  initial  development,  data  previously  collected  in  the  SAAC  were  analyzed  by 
ISAAC.  ISAAC  output  was  then  critiqued  by  the  SME,  who  suggested  corrections, 
additions,  and  refinements  to  the  system.  ISAAC  was  further  developed  and  refined  through 
repeated  iterations  of  SAAC  data  analysis  and  SME  critique  of  ISAAC  output.  A  more 
detailed  description  of  ISAAC  development  can  be  found  in  Thomas,  Obermayer, 
Raspotnik,  and  Waag,  1992. 

Upon  completion  and  SME  verification  of  ISAAC,  analyses  were  conducted  to 
validate  the  expert  system.  Two  tests  were  established  to  validate  the  system:  'Die  first 
specified  the  degree  to  which  the  SME  followed  his  own  air  combat  rules  as  depicted  by 
ISAAC.  The  second  test  was  designed  to  determine  if  air  combat  winners  followed  ISAAC 
rules  more  often  than  did  losers,  and  if  pilots  with  considerable  fighter  experience  were  more 
likely  to  adhere  to  ISAAC  rules  than  were  pilots  with  little  experience  in  fighter  aircraft. 


Method 

A  database  containing  results  of  500  simulated  air  combat  sorties  flown  in  the  SAAC 
was  made  available  for  analysis  purposes.  The  SME  had  flown  four  engagements  each 
against  125  Air  Force  pilots,  whose  experience  in  fighter  aircraft  varied  from  30  to  2,500 
hours.  Each  engagement  started  with  combatants  in  a  neutral  configuration,  either  line 
abreast  or  head  on.  Engagements  concluded  when  one  pilot  achieved  a  "kill"  or  three 
minutes  time  had  elapsed. 

Procedure  1 

The  first  study  was  conducted  to  determine  how  closely  the  SME  followed  his  own 
rules  as  depicted  by  ISAAC.  Thirty  engagements  were  selected  at  random  from  the  SAAC 
database,  with  the  provisions  that  all  were  line-abreast  set-ups,  where  the  SME  won,  lost, 
and  drew  in  each  of  ten  sorties.  These  30  engagements  were  analyzed  by  ISAAC  to 
determine  the  degree  to  which  the  SME  and  his  opponents  followed  the  air  combat  rules 
as  specified  by  the  system.  The  frequency  with  which  pilots  followed  each  of  ISAAC’S  rules 
was  computed  for  each  engagement  flown  by  each  pilot  within  the  categories  of  win,  draw, 
and  lose. 

Procedure  2 


The  second  study  was  designed  to  investigate  the  effect  of  adherence  to  ISAAC  rules 
on  the  probability  of  success  in  air  combat  and  to  determine  if  ISAAC  could  discriminate 
pilots  with  high  fighter  experience  from  those  with  low  fighter  experience.  One-himdred- 
sixty  engagements  were  selected  from  the  SAAC  database.  These  ^jvere,  the  four 
engagements  flown  by  each  of  the  20  pilots  with  the  most  hours  in  fighters  (X  =  1489  hours) 
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and  the  four  engagements  flown  by  each  of  the  20  pilots  with  the  least  hours  in  fighters  (X 
=  56  hours).  The  SME  was  the  opponent  in  all  160  sorties. 


Results 

Table  1  shows  the  frequency  and  the  proportion  of  opportunities  that  the  SME  and 
his  opponents  correctly  followed  ISAAC  rules  as  a  function  of  engagement  outcome.  For 
example,  when  the  SME  won,  he  followed  ISAAC  rules  in  89%  of  the  opportunities 
available.  His  behavior  in  air  combat  more  closely  approximated  the  recominendations  of 
ISAAC  than  did  that  of  his  opponents  (79%  versus  66%).  The  SME  and  his  opponent’s 
behavior  also  matched  the  rules  of  ISAAC  more  often  when  they  won  (83%)  than  when  they 
lost  (67%)  in  simulated  air  combat. 


Table  1.  Frequency  and  Proportion  of  Opportunities  that  the  SME  and  His  Opponents 

Correctly  Followed  ISAAC  Rules 


Win 

Draw 

Ixiss 

.89 

.78 

.74 

SME 

(138/155) 

(324/416) 

(124/167) 

.77 

.65 

.60 

Opponent 

(98/128) 

{1S4I2S2) 

(96/161) 

.83 

.73 

.53 

These  data  were  subjected  to  a  SAS  logit  analysis  to  test  for  main  effects  and 
interactions.  The  resulting  Wald  statistics  indicated  that  the  SME  followed  ISAAC  rules 
significantly  more  often  than  did  his  opponents  (x^  =  27.90,  p<  .00005),  and  that  a  more 
positive  engagement  outcome  (win)  resulted  in  better  pilot  adherence  to  ISAAC  rules  than 
did  a  less  satisfactory  outcome,  such  as  a  loss  ~  20.47,  p  <.00005).  The  pilot-by¬ 
outcome  interaction  was  not  sigmficant. 

Table  2  shows  the  frequency  and  proportion  of  opportunities  that  high-time  and  low¬ 
time  fighter  pilots  followed  the  air  combat  rules  depicted  by  ISAAC  as  a  function  of 
engagement  outcome.  As  indicated  by  the  table,  pilots  who  won  engagements  were  more 
likely  to  follow  ISAAC  rules  (71%)  than  pilots  who  did  not  (53%).  Positive  engagement 
outcome  (i.e.,  win)  again  resulted  in  better  pilot  adherence  to  ISAAC  rules  than  did  less 
positive  outcome  {xl  =  35.03,  p<  .00005).  There  was,  however,  no  significant  difference  in 
the  frequency  with  which  rules  were  followed  by  high-time  versus  low-time  pilots  (%,  =-03). 
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The  outcome-by-pilot  interaction  was  significant  (x^  =  14.24,  p<  .0008),  ^  indicated  by  the 
fact  that  low-time  pilots  followed  ISAAC  rules  slightly  more  often  when  they  lost  than  when 
they  tied  in  simulated  combat. 

Table  2.  Frequency  and  Proportion 

ISAAC  Rules 

of  Opportunities  that  Pilots  Correctly  Followed 

Win 

Draw 

Loss 

.69 

High-Tlme 

(185/269) 

.61 

(442/730) 

.52 

.57 

(626/1215) 

.81 

Low-Time 

(43/53) 

.53 

(376/703) 

.55 

.55 

(843/1535) 

.71 

.57 

.53 

In  general,  SME  performance  in  simulated  air  combat  was  superior  to  that  of  pilots 
he  opposed.  Table  3  indicates  the  number  and  proportion  of  wins,  draws,  an^d  losses 
achieved  by  high  and  low-time  pilots  who  opposed  the  SME.  As  indicated  by  the  table, 
these  pilots  defeated  the  SME  in  only  12%  of  the  engagements,  whereas  they  lost  m  67% 
of  the  simulated  air  combat. 


Table  3.  Pilots’  Number  and  Proportion  of  Wins,  Losses,  and  Draws  Against  the  SME 


Win  _ Draw 

14  16 

High-Time 

.175  .200 


Loss 

50 

.625 


5 

Low-Time 

.063 


17 

57 

,217 

.722 

.12 


.21 


.67 


i 
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Discussion 


The  expertise  of  the  SME  was  evidenced  by  the  fact  that  he  won  67%  of  his 
engagements  against  the  high-time  and  low-time  pilots  while  losing  only  12%  of  his  sorties. 
As  expected,  the  SME  followed  his  own  rules,  as  depicted  by  ISAAC,  more  consistendy  man 
did  other  pilots.  The  SME’s  behavior  matched  that  prescribed  by  ISAAC  m  nearly  80%  ot 
the  available  opportunities.  This  is  impressive  when  one  considers  that  the  opponents  were 
attempting  to  prevent  the  SME  from  following  some  of  the  rules.  For  example,  when 
offensive,  ISAAC  states  that  the  pilot  should  continue  to  point  toward  the  bogey  and  to 
reduce  angle  off  the  tail  (AOT).  In  this  circumstance,  the  opponent  tries  to  mcrease  AUl 
and  prevent  the  fighter  from  pointing  at  him. 

All  pilots,  including  the  SME,  followed  ISAAC  rules  more  firequendy  when  they  won 
simulated  air  combat  than  when  they  lost.  This  result  was  consistently  significant  across  high 
and  low-time  pilots,  and  across  the  initial  conditions  of  line  abreast  and  head  on,  suggestmg 
that  adhering  to  ISAAC  rules  increases  the  likelihood  of  winning  air  combat.  High-time 
pilots  did  not  adhere  to  ISAAC  rules  more  frequently  than  did  low-time  pilofr.  However, 
variables  such  as  recency  of  flying  time  or  combat  training  were  not  controlled  m  this  sample 

of  pilots. 

A  more  detailed  analysis  of  ISAAC  output  revealed  a  few  rules  that  were  followed 
much  more  consistently  by  the  SME  than  by  other  pilots.  The  SME  was  more  successfd  at 
placing  the  bogey  on  the  PVS  immediately  after  the  head-on  passes,  and  then  maxing  Os  . 
This  allowed  the  SME  to  turn  faster  than  the  bogey  and  to  obtain  a  nose-position  advantage 
at  the  subsequent  head-on  pass.  The  nose-position  advantage  afforded  the  SME  more  time 
offensive  than  his  opponent,  increasing  the  likelihood  of  a  successful  missile  or  gun  shot. 

With  minor  enhancements,  ISAAC  will  be  used  as  a  performance  measurement 
system,  capable  of  providing  diagnostic  feedback  to  pilots  practicing  Ivl  air  combat  in  eit  er 
the  SAAC  or  on  instrumented  ranges. 
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